Skip to content

feat(power): validate multinode GPU energy metrics / 校验多机 GPU 能耗指标 - #2437

Open
edwingao28 wants to merge 5 commits into
wenyao/validated-single-node-powerfrom
wenyao/power-pr3-multinode-consumption
Open

feat(power): validate multinode GPU energy metrics / 校验多机 GPU 能耗指标#2437
edwingao28 wants to merge 5 commits into
wenyao/validated-single-node-powerfrom
wenyao/power-pr3-multinode-consumption

Conversation

@edwingao28

Copy link
Copy Markdown
Collaborator

Stacked on #2323 — base is wenyao/validated-single-node-power. After #2323 merges this will rebase and retarget to main. Only the last 3 commits are new.

What this does

Consumes the srt-slurm dcgm-power artifact package (v1 wire contract: power/{manifest.json, samples.csv, windows/*.json}) for disaggregated multinode runs, re-validates it independently, and patches energy metrics into the per-concurrency agg JSON:

  • the 8 whole-deployment metrics from feat(power): validate single-node GPU energy metrics / 校验单节点 GPU 能耗指标 #2323 (power_valid, avg_power_w, avg_total_gpu_power_w, total_gpu_energy_j, joules_per_successful_query, joules_per_input_token, joules_per_output_token, joules_per_total_token)
  • 4 role metrics: prefill_gpu_energy_j, decode_gpu_energy_j, prefill_joules_per_input_token, decode_joules_per_output_token. Role energy is board-level energy over the full serving window. It is not kernel-phase energy.

Trust model: the producer's stored publication_valid is never trusted. The consumer recomputes validity from the persisted bytes, mirroring srt-slurm's own offline validator (identity sextet, strict sample parsing, per-device UUID 1:1, timestamp monotonicity, window status invariants, fixed 3.0 s max sample gap, window bracketing, stored-evidence equality). It then cross-checks the stored verdict; any disagreement is producer_verdict_mismatch and power_valid=0. The producer git commit must equal the POWER_PRODUCER_SHA pin in every mode. The workspace result is bound to its measurement window by canonical-JSON SHA256 against the original at LOGS/<result_path> — the launcher renames results when copying, so names can't be compared.

Default mode is best-effort: invalid telemetry publishes power_valid=0 plus a validation sidecar and never fails the benchmark. REQUIRE_POWER=1 fails the job after the audit artifacts exist.

Validated against a real GB200 artifact

Canary run: https://github.com/SemiAnalysisAI/InferenceX/actions/runs/30587402515 (Qwen3.5-397B FP8, 1P1D TP4, concurrency 4, producer pin 6609d46a4c74ed66fc2f7014d7e790efe1d23bde).

  • srt-slurm's offline validator at the pinned commit: exit 0, publication_valid: True recomputed.
  • This consumer replayed offline on the same package: power_valid=1, avg 343.72 W per GPU (2749.74 W across 8), total_gpu_energy_j=189898.14 over the 69.06 s window, prefill 73214.34 J + decode 116683.80 J (sums exactly).
  • An independent stdlib-only trapezoid recompute from raw samples.csv agrees to 2e-10 relative error.
  • The replay caught one real bug before merge: GB200 1P1D launches as a plain 2-node Slurm job, so every device reports het_group=None. The topology gate now enforces het-group distinctness only when het groups are present. Het-job deployments (e.g. GB300) keep the strict check.

Known deltas (documented, left unchanged)

Tests

  • utils/test_aggregate_power_multinode.py — 22 tests. A self-consistent package builder, then every contract gate tampered one at a time; each rejection asserted in both modes.
  • utils/test_process_result.py::TestMultinodePower — 5 wiring tests through the real subprocess path.
  • .github/workflows/test-process-result.yml now collects the new suite and triggers on the new files.

中文:

堆叠在 #2323——base 为 wenyao/validated-single-node-power#2323 合并后会 rebase 并改回 main,只需 review 最后 3 个 commit。

本 PR 消费 srt-slurm 多机 dcgm-power artifact(v1 wire contract),向 agg JSON 写入 8 个整机指标和 4 个 role 指标。prefill/decode 能耗是整个服务窗口的板级能耗,不是算子阶段能耗。

信任模型:不信任 producer 存储的 publication_valid。消费端从持久化字节独立复算(与 srt-slurm 离线校验器同一套检查),再与 stored verdict 交叉校验,不一致即 producer_verdict_mismatch。producer SHA 必须命中 POWER_PRODUCER_SHA pin。workspace 结果与 window 通过 canonical JSON SHA256 绑定(launcher 拷贝时会改名,不能比文件名)。默认 best-effort,无效遥测只记 power_valid=0 不失败;REQUIRE_POWER=1 才严格失败。

已用真实 GB200 canary artifact 离线验证(数据见上),并借此在合并前修掉一个真实 bug:GB200 1P1D 是普通两节点 Slurm 作业,所有设备 het_group=None,拓扑门限现在只在 het group 存在时强制互异(GB300 het 作业仍严格)。

单元测试 22 个(逐一破坏每个 contract gate,双模式断言)+ process_result 接线测试 5 个;test-process-result workflow 已接入新文件。

中文:新增多机 srt-slurm 电力 artifact 消费模块:严格镜像 producer 合约做独立复算(不信任 stored publication_valid),校验 producer SHA pin、拓扑、样本身份与 window 绑定,梯形积分输出 role 级与全局能耗指标。
中文:多机分支接入 aggregate_power_multinode 消费钩子(best-effort 镜像单机路径,REQUIRE_POWER=1 才失败),internal-error 回退同步清理 role 指标键。
中文:benchmark-multinode-tmpl 新增 require-power/power-producer-sha 输入、REQUIRE_POWER/POWER_PRODUCER_SHA 环境与 power audit bundle 上传;e2e-tests 仅向 multi-node 泳道透传;test-process-result 接入多机模块与测试。
@claude

claude Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Claude finished @edwingao28's task in 4m 43s —— View job


Review: multinode GPU power consumption

  • Gather context (diff vs wenyao/validated-single-node-power, 3 new commits)
  • Review utils/aggregate_power_multinode.py (validator + integration)
  • Review utils/process_result.py wiring
  • Review workflow changes
  • Review tests (CI Test Process Result is green on the head commit)

Summary: One 🟡 warning posted inline — when integration overflows to inf, the sidecar's per_gpu_energy_j is serialized as bare Infinity (Python-only, spec-invalid JSON), so strict parsers like jq can't read the audit artifact in exactly the failure case it exists to explain. No blocking issues: I traced the crash-candidate paths (uuid_label[device.key], _bracketing_sequencepairwise(None), duration_s division, token/query denominators) and each is unreachable when audit.reasons is empty given the device/coverage/window gates plus _load_benchmark_data's end > start and positive-count contracts; the broad except Exception in process_result.py backstops the rest. The verdict cross-check, content-hash result binding, and het-group-optional topology gate all look correct, and the tamper-per-gate test design covers both modes well.

@edwingao28

Copy link
Copy Markdown
Collaborator Author

Validated in CI on real hardware, strict mode (REQUIRE_POWER=1):

  • GB200 (run 30618706258): power_valid=1, 340.51 W/GPU (2724.06 W total), 193,404 J over the 70.998 s window, prefill 74,911 J + decode 118,493 J. Power fields propagate into the aggregated rollup.
  • GB300 (run 30663050396): power_valid=1, 349.31 W/GPU, 172,800 J over 61.836 s, prefill 68,070 J + decode 104,729 J. Exporter runs on port 19401 — 9401 is already bound by a cluster-level exporter on im-gb300 nodes.
  • Both deployments launch as plain 2-node Slurm jobs (het_group=None on every device), so the None-het topology path in this PR is the one real hardware exercises.
  • Both runs re-verified offline: stored vs recomputed producer verdicts agree, producer SHA matches the pin, role energies sum exactly to the total.

Comment thread utils/aggregate_power_multinode.py Outdated
Comment thread utils/process_result.py
edwingao28 and others added 2 commits July 31, 2026 17:31
中文:溢出的 per-GPU 能量此前以 Infinity 字面量写入 sidecar,严格 RFC 8259 解析器(jq/JS/Go)会拒收;改为 null 并在溢出测试中断言 sidecar 严格可解析。
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

1 participant